Skip to content

feat(desktop): add WebKit renderer recovery ladder for Linux - #3271

Open
wpfleger96 wants to merge 5 commits into
mainfrom
duncan/webkit-recovery-ladder
Open

feat(desktop): add WebKit renderer recovery ladder for Linux#3271
wpfleger96 wants to merge 5 commits into
mainfrom
duncan/webkit-recovery-ladder

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Jul 28, 2026

Copy link
Copy Markdown
Member

Some Linux GPU and compositor combinations crash the WebKit web process during startup, leaving Buzz with an invisible window and no way out. WebKit memoizes each of its renderer environment variables exactly once per process (AcceleratedBackingStore.cpp, std::call_once), so a safer configuration can only ever be applied by launching a fresh process — this recovery is built out of explicit relaunches, not runtime toggles.

Closes #2338

Scope

In: crash-driven recovery, --safe-rendering, --reset-rendering-mode.

Out: automatic no-paint detection. A window that never paints but also never crashes is deliberately not detected — no available signal distinguishes it from a healthy window, so that track stays gated on validation against affected hardware.

How it works

On a Crashed web-process termination inside the startup window, the app runs one explicit handoff:

prepare the record atomically
  → release the single-instance name
  → prove the name is free on the session bus
  → spawn a Command with an exact environment
  → the child claims the prepared record
  → the parent exits

tauri::App::request_restart cannot serve here — it has no child-environment parameter and skips the single-instance name release, so every child born from it dies as a duplicate. std::env::set_var is equally out: unsound in a process with live GTK and Tokio threads, and WebKit would have memoized the variable already.

Spawn success and a zero exit code are both worthless as survival signals, since a child that loses the name forwards its argv and exits 0. Only the child's claim receipt, correlated against the name's owner, proves a handoff happened.

Releasing the name is irreversible — the single-instance plugin cannot be re-registered — so the handoff outcome distinguishes a refusal raised before the release from one raised after. Every preflight that can fail, including resolving the binary to re-execute, runs before the release; a refusal there leaves the app running with its name. A refusal after it (the name turned out not to be free, or spawn failed) exits rather than lingering as a Buzz that no longer owns the name. The eligible-crash path is one-shot, and a recovery child that loses its claim exits before Tauri is built rather than competing for the name against the episode's true owner.

Episode state

A durable record moves prepared → started → owned → confirmed, with exclusive (token, generation) claims taken via create_new (O_EXCL). Claim files are never deleted — a stale claim is inert evidence that a generation already ran.

An exclusive claim only serializes the creation of one claim file, which is not enough on its own: reading the record, deciding from it, and writing the next phase are three operations. So every transition runs under an advisory flock on a lock file that is a sibling of the state directory (a lock inode inside it would be unlinked by --reset-rendering-mode from under a concurrent holder). flock rather than an O_EXCL lockfile because the kernel releases it on process death — a lockfile left behind by a process that crashed mid-transition is exactly the failure this feature exists to handle.

Serializing writers is still not sufficient, because a boot decision is taken from a read and acted on after a D-Bus round trip, so the record can move in between. Every durable change therefore goes through one compare-and-transition primitive that re-reads the record under the caller's lock and writes only if it still matches the state the caller decided from — otherwise the write is refused and boot re-decides from what is actually there. A concurrent --reset-rendering-mode cannot be undone by a retry decided before it, and a delayed version-mismatch discard cannot delete a newly prepared episode. A launch that is out-raced repeatedly stands down and runs its launched profile untracked, which writes nothing.

Because the expected source names a phase, the receipt chain is exact rather than merely ordered: owned requires started and confirmed requires owned, so a confirmed that skips a failed owned write is rejected instead of accepted as "forward". The same comparison rejects a receipt from a superseded episode, so a confirmed racing a crash handoff cannot resurrect the episode the crash just disproved. exhausted is a separately validated terminal write against the whole attempt, and nothing leads out of it. Ownership is correlated against the bus (GetConnectionUnixProcessID, unique name, GetId) as absent | correlated | mismatched | uncorrelatable; an owner that exists but cannot be tied to the record is a handoff failure, never a free name. owned receipts are bound to the bus id, because a unique name like :1.4 only identifies a connection on the bus that issued it.

Tier table

Tier Name Variables AppImage Native
0 full-gpu
1 shm-transport WEBKIT_DMABUF_RENDERER_FORCE_SHM=1
2 cpu-raster + WEBKIT_SKIA_ENABLE_CPU_RENDERING=1
3 no-accel WEBKIT_DISABLE_DMABUF_RENDERER=1, WEBKIT_SKIA_ENABLE_CPU_RENDERING=1
4 x11-fallback + GDK_BACKEND=x11

WEBKIT_DISABLE_COMPOSITING_MODE is owned (a user value opts out) but no tier sets it, pending the 3-var vs 4-var isolation on #2643.

One crash advances exactly one rung and no rung is ever repeated, so the tier count is the per-package attempt cap — no separate counter can drift from it.

Environment ownership

Package-scoped. AppImage owns the 4 WEBKIT_* variables; native additionally owns GDK_BACKEND, which on AppImage is a package invariant re-exported by linuxdeploy's GTK AppRun hook before the binary starts and therefore never a user signal.

Any genuine user assignment of an owned variable — any value, including 0 and empty — disables recovery wholesale with a log line. --safe-rendering combined with such an assignment is refused as a fatal pre-Tauri error naming the conflicting variables, rather than guessed either way: honouring the flag would overwrite configuration the user typed, and honouring the environment would silently ignore a rescue flag from a user whose app will not start. Recovery children are tagged BUZZ_RENDER_EPISODE / BUZZ_RENDER_GENERATION / BUZZ_RENDER_PROFILE so they never reinterpret an injected profile as user configuration.

AppImage verification

Built Buzz_0.4.26_aarch64.AppImage and ran it under xvfb-run dbus-run-session. The crash edge is driven by SIGKILL on the WebKit web process inside the eligibility window — the same Crashed termination the reporters hit.

Crash-driven walk from a clean state — all four AppImage rungs, then terminal:

render-recovery: handed off to shm-transport as pid 135924
render-recovery: handed off to cpu-raster as pid 135998
render-recovery: handed off to no-accel as pid 136102
render-recovery: ladder exhausted on appimage at tier 3 (no-accel); relaunch with --safe-rendering to force the safest profile

Exactly one survivor, and it is the D-Bus name owner:

SURVIVOR COUNT: 1
owner unique name: :1.33
owner pid: uint32 136106

Its environment is the tier and nothing inherited:

BUZZ_RENDER_EPISODE=8bed27ed-f561-4c3a-85ab-16edd42e8393
BUZZ_RENDER_GENERATION=0
BUZZ_RENDER_PROFILE=no-accel
GDK_BACKEND=x11                        # AppRun invariant, not a tier value
WEBKIT_DISABLE_DMABUF_RENDERER=1
WEBKIT_SKIA_ENABLE_CPU_RENDERING=1

The record ends exhausted at tier 3 naming that pid, with one claim file per generation and the lock file a sibling of the state directory:

render-recovery.lock
render-recovery/episode.json
render-recovery/claims/8bed27ed-f561-4c3a-85ab-16edd42e8393-g0

Flags on the same build: --safe-rendering forces no-accel and leaves no state directory behind; WEBKIT_DISABLE_DMABUF_RENDERER=0 in the user's environment disables recovery wholesale and persists nothing; --safe-rendering under that same assignment exits 1 before Tauri with the conflict named and nothing persisted; --reset-rendering-mode clears the record and exits without launching, and reports accurately on a second run.

Layout

desktop/src-tauri/src/render_recovery/profiles (package scoping + tier table), state (durable record, exclusive claims, the compare-and-transition primitive), episode (attempt identity + pure ladder rules), dbus (name observation), classify (record + observation → decision), launcher (the explicit relaunch), cli (flags), session (live process, with session::handoff for prepare/release/spawn), wiring (Tauri plugin). 65 unit tests cover the three generation races, injected deaths at all four points, the per-package cap, the version-scoped crash-free-startup fact, and the exact receipt chain including every phase skip.

The races that need real contention are driven rather than simulated. Two same-generation children race as two real processes through a filesystem barrier, so their pids differ and the durable receipt can be checked against the identity of the one that won. A stale generation claiming while a newer one is prepared, a confirmation against a crash handoff, two termination callbacks, and a rollback against a later episode are driven from two threads through a barrier with the state lock held across the window each one needs, and each asserts the second writer actually blocks. A reset racing a decision taken before it, and a delayed version-mismatch discard against a fresh episode, assert the write is refused and the record is left intact.

lib.rs gains 37 lines: the boot call before Tauri is built, the four boot outcomes it must act on, and plugin registration after the single-instance plugin.

Related: desktop/scripts/fix-appimage.sh carries no rendering environment exports and is unchanged here — the ladder owns those variables exclusively.

Some Linux GPU and compositor combinations crash the WebKit web process
during startup, leaving Buzz with an invisible window and no way out
(#2338, #2643). WebKit memoizes each renderer environment variable once
per process, so a safer configuration can only ever be applied by
launching a fresh process — the recovery is therefore built out of
explicit relaunches rather than runtime toggles.

On a crashed web process inside the startup window, the app prepares a
durable episode record, releases the single-instance D-Bus name, proves
the name is free, spawns a child with the next tier's exact environment,
and exits. The child claims the record exclusively and confirms once it
has started cleanly; a confirmed tier is reused on later launches.

Recovery is crash-driven only. A window that never paints but also never
crashes is not detected, because no available signal distinguishes it
from a healthy window.

Co-authored-by: Will Pfleger <[email protected]>
Signed-off-by: Will Pfleger <[email protected]>
@wpfleger96
wpfleger96 requested a review from a team as a code owner July 28, 2026 05:16
…lease boundary

Two safety invariants in the recovery ladder were not enforced.

A refusal raised after the single-instance name was released left the
parent running without the name and unable to take it back, because the
launcher's Result<(), Refusal> could not express which side of that
irreversible boundary the refusal came from — and binary resolution, the
one preflight most likely to fail, happened inside the launcher. Both
refusal sides are now distinct outcomes, every preflight runs before the
release, and a post-release refusal exits instead of lingering.

The exclusive claim serialized claim-file creation, not record
transitions: read-validate-write ran as three unsynchronized steps, so a
delayed generation could stamp a stale receipt over a newer episode, a
duplicated crash callback could spawn a second child, and a claim loser
could reach the single-instance name ahead of the true owner. Every
transition now runs under a crash-releasing advisory lock on a file
outside the cleared directory, receipts must name the current
(token, generation) and move the phase forward, the eligible-crash path
is one-shot, and claim losers exit before Tauri is built.

--safe-rendering against a user-set owned variable was silently dropped;
it is now a fatal pre-Tauri error naming the conflicting assignments.

Co-authored-by: Will Pfleger <[email protected]>
Signed-off-by: Will Pfleger <[email protected]>
Boot decided from a record read outside the transaction lock and then
mutated under it without re-checking that record was still current, so a
decision could outlive the state it rested on: a concurrent
--reset-rendering-mode could be undone by a stale retry recreating a
prepared episode, and a delayed version-mismatch discard could delete a
newly prepared current-version episode.

Store::transition is now the only way anything durable changes. It
re-reads the record under the caller's lock and writes only if the record
still matches the state the caller decided from; a refusal makes boot
re-decide from what is actually there rather than proceed. Because the
expected source names a phase, the receipt chain is exact for free --
owned requires started, confirmed requires owned -- replacing a
rank-based ordering that accepted skips.

The prepare/release/spawn sequence moves to session::handoff, which keeps
both files well clear of the size gate.

Co-authored-by: Will Pfleger <[email protected]>
Signed-off-by: Will Pfleger <[email protected]>
The two tests that inject a competing write from inside the spawn window
shared one static path slot. The launch edge is a bare fn pointer and
captures nothing, so the path has to reach it through a static -- but with
one slot the tests overwrite each other's path whenever the harness runs
them in parallel, which fails the assertions in CI while passing under a
luckier local ordering.

Verified by reintroducing the shared slot: 6/6 runs fail. With a slot per
edge, 8/8 pass.

Co-authored-by: Will Pfleger <[email protected]>
Signed-off-by: Will Pfleger <[email protected]>
Compare-and-transition refuses a write whose source record moved, and the
launch is then supposed to retake the decision against what is actually
there. Nothing proved the retake happens: a launch that gave up on the
refusal would have passed every existing test.

The new case parks a launch on the state lock mid-handoff, replaces the
record it classified with a confirmed tier 3, and asserts the handoff
lands at tier 3 -- a tier only a second classification could have chosen.
Verified by capping the reconcile loop at one attempt: this test is the
only one that fails.

Co-authored-by: Will Pfleger <[email protected]>
Signed-off-by: Will Pfleger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant